Augmented Partial Mutual Learning with Frame Masking for Video Captioning

Recent video captioning work improves greatly due to the invention of various elaborate model architectures. If multiple models are combined into a unified framework not only by simple more ensemble, and each can benefit from other, final might be boosted further. Jointly training have been explored in previous works. In this paper, we propose novel Augmented Partial Mutual Learning (APML) method where decoders trained jointly with mimicry losses between different input variations. Another problem is "one-to-many" mapping which means that one identical mapped caption annotations. To address problem, an annotation-wise frame masking approach convert "one-to-one" mapping. The experiments performed on MSR-VTT MSVD datasets demonstrate our proposed algorithm achieves state-of-the-art performance.